Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(task db): implement a task DB #208

Merged
merged 52 commits into from
Jul 2, 2024
Merged

feat(task db): implement a task DB #208

merged 52 commits into from
Jul 2, 2024

Conversation

mratsim
Copy link
Contributor

@mratsim mratsim commented May 15, 2024

Overview

When dealing with provers we need to be able to persist state of proving requests in case of network failure or hardware restart or cancellations.

In particular for Bonsai, without storing state, we would be unable to know how many tasks are in the queue, risking waiting in vain if there were none, or overloading the queue by mistake.

Usage

The DB introduced in this PR should be used either in front of Raiko, from taiko-geth or taiko-reth. Or after raiko. It can be either integrated in an existing product or as a microservice, for example as a daemon or a server.

Note

This PR currently has no doc as I need to leave it 95% finished due to personal reasons.

For @petarvujovic98, as discussed please review the error model so that it fits Raiko with anyhow, thiserror, ....

I might not be able to update it for several days so feel free to take over.

API

For now this provides

  • A way to enqueue a new task
  • A way to update the status(es) of an existing task, including caching a proof and changing proof providers.
  • A way to retrieve a cached proof
  • A way to get a task status. This returns a vector (provider, status, timestamp) as a task might be in network failure say in Bonsai, and we switch to local proving and it's in-progress.

This will need a way to clear the GuestInput/Payload with "older than Duration" parameter, default 18 days (EIP-4844 blobs expiry) as those are several megabytes.

@petarvujovic98 petarvujovic98 changed the title feat(task db): Implement a task DB feat(task db): implement a task DB May 20, 2024
…tion (#296)

* impl a mem db for easy integration

Signed-off-by: smtmfft <smtm@taiko.xyz>

* fix clippy and unit test

Signed-off-by: smtmfft <smtm@taiko.xyz>

* fix fmt

Signed-off-by: smtmfft <smtm@taiko.xyz>

---------

Signed-off-by: smtmfft <smtm@taiko.xyz>
petarvujovic98 and others added 6 commits July 2, 2024 11:00
* enable sqlite db by feature

Signed-off-by: smtmfft <smtm@taiko.xyz>

* debug lifetime

Signed-off-by: smtmfft <smtm@taiko.xyz>

* resolve lifetime issue and make all tests pass

Signed-off-by: smtmfft <smtm@taiko.xyz>

* refactor(task_db): simplify structure for sqlite and use cached statements

* feat(task_db): abstract task db implementation into wrapper

* fix(task_db): add await to test call

* fix(task_db): fix import declaration

* fix(task_db): add async and mutable variables

* fix(host): fix task manager usage

* fix(task_db): fix test for async

* Update Cargo.toml

use in-mem as default.

---------

Signed-off-by: smtmfft <smtm@taiko.xyz>
Co-authored-by: smtmfft <smtm@taiko.xyz>
Co-authored-by: smtmfft <99081233+smtmfft@users.noreply.github.com>
@petarvujovic98 petarvujovic98 added this pull request to the merge queue Jul 2, 2024
Merged via the queue into main with commit 48ea079 Jul 2, 2024
12 checks passed
@petarvujovic98 petarvujovic98 deleted the tasks_db branch July 2, 2024 18:01
Champii pushed a commit to Champii/raiko that referenced this pull request Jul 2, 2024
* task-manager: dump fight vs sqlite

* task-manager: SQL tables+views+triggers success - but arguments passed in execute are 'NULL'

* task-manager: passing enqueue_task tests

* task-manager: cleanup - ease copy pasting to SQL script, Registered status, persistent views, remove debug print

* task-manager: add DB size query

* task-manager: id_proof is unneeded + prettify queries

* task-manager: change DB schema - allow multiple provers and status for same task in case of failures-retry

* task-manager: allow task updates

* task-manager: retrieve cached proofs from DB

* task-manager: add status check

* task-manager: add progress reports

* chore(task_manager): Run cargo fmt

* feat: address small lints

* feat(task-manager): use result type with thiserror

* chore(task-db): fix typos

* refactor(task-manager): clean up tests

* fix(docker): unignore task manager

* [WIP](task_manager): write initial task handler stubs

* chore(task_manager): run cargo fmt

* [WIP](task_manager): write status and proof get handlers

* refactor(host): use merge instead of nest

* chore(format): format workflow files

* chore(deps): use consistent dependency style

* chore(host): rename tx to task_channel

* [WIP](task_manager): add initial submit logic

* chore(clippy): remove unused parameter

* chore(clippy): remove unused imports

* refactor(core): add copy trait to proof types

* feat(task_manager): simplify db and adapt tests

* fix(clippy): fix dereference issue

* [WIP]: handle proof request by worker and update task status

* [WIP]: add block fetching and initial blockhash getting for submit

* [WIP]: handle task creation, status and proof retrieval

* fix(host): fix route sub-path

* feat(raiko): abstract task manager and impl a mem db for easy integration (taikoxyz#296)

* impl a mem db for easy integration

Signed-off-by: smtmfft <smtm@taiko.xyz>

* fix clippy and unit test

Signed-off-by: smtmfft <smtm@taiko.xyz>

* fix fmt

Signed-off-by: smtmfft <smtm@taiko.xyz>

---------

Signed-off-by: smtmfft <smtm@taiko.xyz>

* fix: throw error instead of panicing on runtime checks

* fix(core,task_manager): add custom ensure and require fns

* feat(task_db): sqlite and in memory abstraction (taikoxyz#301)

* enable sqlite db by feature

Signed-off-by: smtmfft <smtm@taiko.xyz>

* debug lifetime

Signed-off-by: smtmfft <smtm@taiko.xyz>

* resolve lifetime issue and make all tests pass

Signed-off-by: smtmfft <smtm@taiko.xyz>

* refactor(task_db): simplify structure for sqlite and use cached statements

* feat(task_db): abstract task db implementation into wrapper

* fix(task_db): add await to test call

* fix(task_db): fix import declaration

* fix(task_db): add async and mutable variables

* fix(host): fix task manager usage

* fix(task_db): fix test for async

* Update Cargo.toml

use in-mem as default.

---------

Signed-off-by: smtmfft <smtm@taiko.xyz>
Co-authored-by: smtmfft <smtm@taiko.xyz>
Co-authored-by: smtmfft <99081233+smtmfft@users.noreply.github.com>

* feat(task_manager): return empty list on key not found

* feat(host,task_manager): add tracing and handle workers

* feat(host): fix response structure

* chore(clippy): remove unused imports

* fix(ci): remove git merge added lines

* fix(task_manager): add blob proof type field

---------

Signed-off-by: smtmfft <smtm@taiko.xyz>
Co-authored-by: Petar Vujović <petarvujovic98@gmail.com>
Co-authored-by: smtmfft <99081233+smtmfft@users.noreply.github.com>
Co-authored-by: smtmfft <smtm@taiko.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants